Skip to content

Make analytics multi-backend, add native solution#1090

Merged
gmarull merged 8 commits intocoredevices:mainfrom
teslabs:analytics-multi-backend
Apr 15, 2026
Merged

Make analytics multi-backend, add native solution#1090
gmarull merged 8 commits intocoredevices:mainfrom
teslabs:analytics-multi-backend

Conversation

@gmarull
Copy link
Copy Markdown
Member

@gmarull gmarull commented Apr 13, 2026

This transforms the analytics service into multi-backend capable. It also adds a native backend option that does not rely on third-party services. Compared to the old native implementation, here we should not rely on any fixed blob layout. Instead, the ELF file can be used to obtain the actual blob layout/fields/sizes, etc. (see the last commit with a tool that shows how to do that).

Needs some testing...!

@gmarull gmarull requested review from ericmigi and jplexer April 13, 2026 19:26
@gmarull gmarull force-pushed the analytics-multi-backend branch 4 times, most recently from fce9770 to 5290603 Compare April 13, 2026 20:01
@ericmigi
Copy link
Copy Markdown
Collaborator

ericmigi commented Apr 13, 2026

just updated eng-dash to match this format. @sjp4 can you add this to mobile pls?

sjp4
sjp4 reviewed Apr 14, 2026
View reviewed changes
Comment thread src/fw/services/common/analytics/native.c Outdated

PBL_ASSERTN(dls_initialized());

s_dls_session = dls_create(DlsSystemTagAnalyticsDeviceHeartbeat, DATA_LOGGING_BYTE_ARRAY,
Copy link
Copy Markdown
Member

@sjp4 sjp4 Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we use a different/new tag for this? The existing tag will be reporting from legacy firmwares with a different format, so maybe we should separate this format

Copy link
Copy Markdown
Member Author

@gmarull gmarull Apr 14, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'm gonna fix this

@gmarull
Copy link
Copy Markdown
Member Author

gmarull commented Apr 14, 2026 •
edited
Loading

just updated eng-dash to match this format. @sjp4 can you add this to mobile pls?

The new mechanism should work as follows:

  1. Add a new eng-dash endpoint where we can start uploading ELF files; this way, you can parse (see example tool): (1) the build ID and (2) the blob layout
  2. Mobile is the dumb part: it should just get the blob and post it to the server
  3. The server can now match the blob ID with any previously uploaded ELF ID. If there's a match, simply take data following blob layout.

This makes sure the backend will always store new metrics without any changes, and that non-released builds will be ignored (no ELF match).

@gmarull gmarull force-pushed the analytics-multi-backend branch 4 times, most recently from 181dc42 to b7d7e25 Compare April 14, 2026 10:22
@gmarull gmarull marked this pull request as ready for review April 14, 2026 11:43
@gmarull gmarull force-pushed the analytics-multi-backend branch from b7d7e25 to 575d3d7 Compare April 14, 2026 12:33
@sjp4
Copy link
Copy Markdown
Member

sjp4 commented Apr 14, 2026

just updated eng-dash to match this format. @sjp4 can you add this to mobile pls?

see mobile branch steve/analytics if you want to try it - i tried testing with this fw branch on asterix but couldn't get it to boot (not sure if it's my setup or... but time for bed right now)

@gmarull
Copy link
Copy Markdown
Member Author

gmarull commented Apr 14, 2026

just updated eng-dash to match this format. @sjp4 can you add this to mobile pls?

see mobile branch steve/analytics if you want to try it - i tried testing with this fw branch on asterix but couldn't get it to boot (not sure if it's my setup or... but time for bed right now)

It should boot now, force pushed a few mins ago

@ericmigi
Copy link
Copy Markdown
Collaborator

ericmigi commented Apr 14, 2026

Plan is to have GitHub CI upload the elf?

@ericmigi
Copy link
Copy Markdown
Collaborator

ericmigi commented Apr 14, 2026

Problem is I don't want to have it auto add new columns in bigquery if we can help it...so we'll still need a manual step somewhere (the new data will be stored in the raw field so we don't lose it. )

@gmarull
Copy link
Copy Markdown
Member Author

gmarull commented Apr 14, 2026

Problem is I don't want to have it auto add new columns in bigquery if we can help it...so we'll still need a manual step somewhere (the new data will be stored in the raw field so we don't lose it. )

We can enable fields we want, but we should process all we get and store it (same as memfault really). We could have a place where to select which detected metrics we want to track, give them a name, etc. It's a more robust approach IMHO, things are decoupled which is great. The blob also contains scaling factors.

@gmarull
Copy link
Copy Markdown
Member Author

gmarull commented Apr 14, 2026

Plan is to have GitHub CI upload the elf?

Yes, same as we do for Memfault

@ericmigi
Copy link
Copy Markdown
Collaborator

ericmigi commented Apr 14, 2026 •
edited
Loading

We can enable fields we want, but we should process all we get and store it (same as memfault really). We could have a place where to select which detected metrics we want to track, give them a name, etc. It's a more robust approach IMHO, things are decoupled which is great. The blob also contains scaling factors.

it's all tracked in a raw field so we don't lose any data but we'll need to add column/to the metrics viewer - it won't do it automatically. I think we're saying the same things.

@gmarull
Copy link
Copy Markdown
Member Author

gmarull commented Apr 14, 2026

We can enable fields we want, but we should process all we get and store it (same as memfault really). We could have a place where to select which detected metrics we want to track, give them a name, etc. It's a more robust approach IMHO, things are decoupled which is great. The blob also contains scaling factors.

it's all tracked in a raw field so we don't lose any data but we'll need to add column/to the metrics viewer - it won't do it automatically. I think we're saying the same things.

That works as soon as old raw can be post processed when new column is added. I think Memfault just inserts columns automatically and adds data, so you can immediately plot/track whatever you want. But you can also archive a metric, and customize a few aspects of it. IMHO Memfault does the right thing here, it's just not great in their inspection capabilities (plots, stats, etc.)...

@ericmigi
Copy link
Copy Markdown
Collaborator

ericmigi commented Apr 14, 2026

ok it's merged and deployed to eng-dash, can you give me a sample file or something that I can test the ingester with?

@gmarull
Copy link
Copy Markdown
Member Author

gmarull commented Apr 14, 2026

ok it's merged and deployed to eng-dash, can you give me a sample file or something that I can test the ingester with?

Just build this branch and take tintin_fw.elf

@gmarull gmarull requested a review from sjp4 April 15, 2026 08:42
sjp4
sjp4 approved these changes Apr 15, 2026
View reviewed changes
Comment thread src/fw/console/prompt_commands.h
Comment thread third_party/memfault/port/include/memfault_platform_config.h
gmarull added 3 commits April 15, 2026 15:25
@gmarull
third_party/memfault/port: add missing include
2c1113e 
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull
fw/services/common/analytics: add metrics definition file
d21e858 
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull
fw/services/common/analytics: convert to multi-backend solution
bb2f885 
So we can use multiple backends at the same time.

Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull gmarull force-pushed the analytics-multi-backend branch 3 times, most recently from ca7f672 to dfe7d0d Compare April 15, 2026 13:53
gmarull and others added 5 commits April 15, 2026 16:01
@gmarull @claude
fw/services/common/analytics: add native analytics backend
59015f8 
Add the native analytics backend that logs heartbeat records via DLS.
Metrics are stored in flat arrays indexed by key enum and serialized
into a packed struct on each heartbeat.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull
fw/services/common/analytics: nativize battery/connectivity metrics
b8e0e3b 
Instead of relying on Memfault. The new native solution keeps
compatibility with Memfault, but one can also extract the same info
using native metrics alone.

Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull @claude
tools: add analytics heartbeat layout extraction script
ab09370 
Add a pyelftools-based script that extracts the native_heartbeat_record
struct layout from ELF DWARF debug info, printing field names, offsets,
sizes, and types. Uses a fast binary search in .debug_str/.debug_info
to locate the right CU, avoiding full DWARF parsing. Useful for building
parsers for the analytics DLS blob.

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull
tests/fw/services/bluetooth: remove stale analytics test
38666bb 
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull
agents: ruff is our Python formatter
2f91510 
Signed-off-by: Gerard Marull-Paretas <gerard@teslabs.com>
@gmarull gmarull force-pushed the analytics-multi-backend branch from dfe7d0d to 2f91510 Compare April 15, 2026 14:01
@gmarull gmarull merged commit 20ca8e6 into coredevices:main Apr 15, 2026
42 checks passed
@gmarull gmarull deleted the analytics-multi-backend branch April 15, 2026 14:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@sjp4 sjp4 sjp4 approved these changes

@ericmigi ericmigi Awaiting requested review from ericmigi

@jplexer jplexer Awaiting requested review from jplexer

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@gmarull @ericmigi @sjp4